Lessons learned from automating tests for an operations support system

We present experience gained in automating tests for an operations support system. A major portion of the effort was devoted to extending a commercial test tool so that testers could easily manipulate GUI objects on two implementations of the application. For this purpose, we developed a test automation library as support infrastructure for writing tests. The challenges and tradeoffs are discussed such as simplicity/complexity for a tester vs. a library developer, hiding/exposing window hierarchy to the tester, providing common methods for different types of GUI objects, transparently manipulating custom GUI widgets, and coping with data-dependent test cases. We discuss the requirements of test code reusability, maintainability, and portability, and describe the solutions we found. In addition, we offer observations about benefits and pitfalls of test automation, our recommendations for maximizing return on investment, and results from automating a variety of tests.